Introduction
- TMLE is a general algorithm for the construction of double robust, semiparametric, efficient substitution estimators. TMLE allows for data-adaptive estimation while obtaining valid statistical inference.
- Although TMLE implemtation uses the G-computation estimand (G-formula). Briefly, the TMLE algorithm uses information in the estimated exposure mechanism P(A|W) to update the initial estimator of the conditional mean E\(_{0}\)(Y|A,W).
- The targeted estimates are then substituted into the parameter mapping. The updating step achieves a targeted bias reduction for the parameter of interest \(\psi(P_{0})\) (the true target parameter) and serves to solve the efficient score equation. As a result, TMLE is a double robust estimator.
- TMLE it will be consistent for \(\psi(P_{0})\) is either the conditional expectation E\(_{0}\)(Y|A,W) or the exposure mechanism P\(_{0}\)(A|W) are estimated consistently. When both functions are consistently estimated, the TMLE will be efficient in that it achieves the lowest asymptotic variance among a large class of estimators. These asymptotic properties typically translate into lower bias and variance in finite samples.(Bühlmann et al., 2016)
- The advantages of TMLE have been repeatedly demonstrated in both simulation studies and applied analyses.(Laan and Rose, 2011)
- The procedure is available with standard software such as the tmle package in R (Gruber and Laan, 2011).
Causal assumptions
Under the counterfactual framework, we have to consider the following assumptions to consider the estimate of the ATE as causal:
CMI or Randomization
(\(Y_{0},Y_{1}\perp\)A|W) of the binary treatment effect (A) on the outcome (Y) given the set of observed covariates (W), where W = (W1, W2, W3, … , Wk).
Positivity
a ϵ A: P(A=a | W) > 0
P(A=1|W=w) > 0 and P(A=0| W = w) > 0 for each possible w.
Consistency or SUTVA:
The observed outcome value, under the observed treatment, is equal to the counterfactual outcome corresponding to the observed treatment for identical independent distributed (i.i.d.) variables.
TMLE flow chart
Source : Mark van der Laan and Sherri Rose. Targeted learning: causal inference for observational and experimental dataSpringer Series in Statistics, 2011 
Data generation
In R we create a function to generate the data with the input number of draws and the output the observed data (ObsData) plus the counterfactuals (Y1, Y0).
The observed data:
1. Y: mortality binary indicator (1 death, 0 alive) 2. A: binary treatment for emergency presentation at cancer diagnosis (1 EP, 0 NonEP)
3. W1: Gender (1 male; 0 female)
4. W2: Age at diagnosis (0 <65; 1 >=65)
4. W3: Cancer TNM classification (scale from 1 to 4)
5. W4: Comorbidities (scale from 1 to 5)
#install.packages("broom")
options(digits=3)
generateData <- function(n){
w1 <- rbinom(n, size=1, prob=0.5)
w2 <- rbinom(n, size=1, prob=0.65)
w3 <- runif(n, min=0, max=4)
w4 <- runif(n, min=0, max=5)
A <- rbinom(n, size=1, prob= plogis(-0.4 + 0.2*w2 + 0.15*w3 + 0.2*w4))
Y <- rbinom(n, size=1, prob= plogis(-1 + A -0.1*w1 + 0.3*w2 + 0.25*w3 + 0.2*w4))
# counterfactual
Y.1 <- rbinom(n, size=1, prob= plogis(-1 + 1 -0.1*w1 + 0.3*w2 + 0.25*w3 + 0.2*w4))
Y.0 <- rbinom(n, size=1, prob= plogis(-1 + 0 -0.1*w1 + 0.3*w2 + 0.25*w3 + 0.2*w4))
# return data.frame
data.frame(w1, w2, w3, w4, A, Y, Y.1, Y.0)
}
set.seed(7858)
ObsData <- generateData(n=1000)
True_Psi <- mean(ObsData$Y.1-ObsData$Y.0);True_Psi
[1] 0.241
Bias_Psi <- summary(lm(data=ObsData, Y~A));Bias_Psi$coef
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.543 0.0238 22.79 6.48e-93
A 0.210 0.0300 6.99 4.98e-12
Data visualization
# DT table = interactive
# install.packages("DT") # install DT first
library(DT)
datatable(head(ObsData, n = nrow(ObsData)), options = list(pageLength = 5))
TMLE implementation
1st step: E\(_{0}\)(Y|A,W)
Estimation of the initial probability of the outcome (Y) given the treatment (A) and the set of covariates (W), denoted as the \(Q_{0}\)(A,W). To estimate \(Q_{0}\)(A,W) we can use a standard logistic regression model:
\(logit[P(Y=1|A,W)]\,=\,\beta_{0}\,+\,\beta_{1}A\,+\,\beta_{2}^{T}\)W.
Therefore, we can estimate the initial probability (as follows: . (1) The predicted probability can be estimated using the Super Learner library implemented in the R package “Super-Learner”6 to include any terms that are functions of A or W (e.g., polynomial terms of A and W, as well as the interaction terms of A and W, can be considered). Consequently, for each subject, the predicted probabilities for both potential outcomes and can be estimated by setting A = 0 and A = 1 for everyone respectively: and,.
Thank you
Thank you for participating in this tutorial.
If you have updates or changes that you would like to make, please send me a pull request. Alternatively, if you have any questions, please e-mail me.
Miguel Angel Luque Fernandez
E-mail: miguel-angel.luque at lshtm.ac.uk
Twitter @WATZILEI
Session Info
devtools::session_info()
References
Bühlmann P, Drineas P, Laan M van der, Kane M. (2016). Handbook of big data. CRC Press.
Greenland S, Robins JM. (1986). Identifiability, exchangeability, and epidemiological confounding. International journal of epidemiology 15: 413–419.
Gruber S, Laan M van der. (2011). Tmle: An r package for targeted maximum likelihood estimation. UC Berkeley Division of Biostatistics Working Paper Series.
Laan M van der, Rose S. (2011). Targeted learning: Causal inference for observational and experimental data. Springer Series in Statistics.
LS0tCnRpdGxlOiAiVE1MRSBzdGVwIGJ5IHN0ZXAiCmF1dGhvcjogJ0J5OiBNaWd1ZWwgQW5nZWwgTHVxdWUgRmVybmFuZGV6LCBtaWd1ZWwtYW5nZWwubHVxdWVAbHNodG0uYWMudWsnCmRhdGU6ICJPY3RvYmVyIDE1dGgsIDIwMTYiCm91dHB1dDogIAogIGh0bWxfbm90ZWJvb2s6CiAgICBjb2RlX2ZvbGRpbmc6IHNob3cKICAgIGhpZ2hsaWdodDogZGVmYXVsdAogICAgI2tlZXBfbWQ6IHllcwogICAgbnVtYmVyX3NlY3Rpb25zOiB5ZXMKICAgIHRoZW1lOiBqb3VybmFsCiAgICB0b2M6IHllcwogICAgdG9jX2Zsb2F0OgogICAgICBjb2xsYXBzZWQ6IG5vCiAgICAgIHNtb290aF9zY3JvbGw6IHllcwogICAgICB0b2NfZGVwdGg6IDMKY3NsOiByZWZlcmVuY2VzL2lzbWUuY3NsCmJpYmxpb2dyYXBoeTogcmVmZXJlbmNlcy9iaWJsaW9ncmFwaHkuYmliCmZvbnQtaW1wb3J0OiBodHRwOi8vZm9udHMuZ29vZ2xlYXBpcy5jb20vY3NzP2ZhbWlseT1SaXNxdWUKZm9udC1mYW1pbHk6ICdSaXNxdWUnCi0tLQoKPCEtLUJFR0lOOiAgU2V0IHRoZSBnbG9iYWwgb3B0aW9ucyBhbmQgbG9hZCBwYWNrYWdlcy0tPgpgYGB7ciBzZXQtZ2xvYmFsLW9wdGlvbnMsIGVjaG8gPSBGQUxTRX0Ka25pdHI6Om9wdHNfY2h1bmskc2V0KGV2YWwgPSBUUlVFLCAKICAgICAgICAgICAgICAgICAgICAgIGVjaG8gPSBUUlVFLCAKICAgICAgICAgICAgICAgICAgICAgIGNhY2hlID0gRkFMU0UsCiAgICAgICAgICAgICAgICAgICAgICBpbmNsdWRlID0gVFJVRSwKICAgICAgICAgICAgICAgICAgICAgIGNvbGxhcHNlID0gRkFMU0UsCiAgICAgICAgICAgICAgICAgICAgICBkZXBlbmRzb24gPSBOVUxMLAogICAgICAgICAgICAgICAgICAgICAgZW5naW5lID0gIlIiLCAjIENodW5rcyB3aWxsIGFsd2F5cyBoYXZlIFIgY29kZSwgdW5sZXNzIG5vdGVkCiAgICAgICAgICAgICAgICAgICAgICBlcnJvciA9IFRSVUUsCiAgICAgICAgICAgICAgICAgICAgICBmaWcucGF0aD0iRmlndXJlcy8iLCAgIyBTZXQgdGhlIGZpZ3VyZSBvcHRpb25zCiAgICAgICAgICAgICAgICAgICAgICBmaWcuYWxpZ24gPSAiY2VudGVyIiwgCiAgICAgICAgICAgICAgICAgICAgICBmaWcud2lkdGggPSA3LAogICAgICAgICAgICAgICAgICAgICAgZmlnLmhlaWdodCA9IDcpCmBgYAoKI0ludHJvZHVjdGlvbgoxLiAqKlRNTEUqKiBpcyBhIGdlbmVyYWwgYWxnb3JpdGhtIGZvciB0aGUgY29uc3RydWN0aW9uIG9mIGRvdWJsZSByb2J1c3QsIHNlbWlwYXJhbWV0cmljLCBlZmZpY2llbnQgc3Vic3RpdHV0aW9uIGVzdGltYXRvcnMuIFRNTEUgYWxsb3dzIGZvciBkYXRhLWFkYXB0aXZlIGVzdGltYXRpb24gd2hpbGUgb2J0YWluaW5nIHZhbGlkIHN0YXRpc3RpY2FsIGluZmVyZW5jZS4gCjIuIEFsdGhvdWdoICoqVE1MRSoqIGltcGxlbXRhdGlvbiB1c2VzIHRoZSBHLWNvbXB1dGF0aW9uIGVzdGltYW5kIChHLWZvcm11bGEpLiBCcmllZmx5LCB0aGUgVE1MRSBhbGdvcml0aG0gdXNlcyBpbmZvcm1hdGlvbiBpbiB0aGUgZXN0aW1hdGVkIGV4cG9zdXJlIG1lY2hhbmlzbSBQKEF8VykgdG8gdXBkYXRlIHRoZSBpbml0aWFsIGVzdGltYXRvciBvZiB0aGUgY29uZGl0aW9uYWwgbWVhbiBFJF97MH0kKFl8QSxXKS4gCjMuIFRoZSB0YXJnZXRlZCBlc3RpbWF0ZXMgYXJlIHRoZW4gc3Vic3RpdHV0ZWQgaW50byB0aGUgcGFyYW1ldGVyIG1hcHBpbmcuIFRoZSB1cGRhdGluZyBzdGVwIGFjaGlldmVzIGEgdGFyZ2V0ZWQgYmlhcyByZWR1Y3Rpb24gZm9yIHRoZSBwYXJhbWV0ZXIgb2YgaW50ZXJlc3QgJFxwc2koUF97MH0pJCAodGhlIHRydWUgdGFyZ2V0IHBhcmFtZXRlcikgYW5kIHNlcnZlcyB0byBzb2x2ZSB0aGUgZWZmaWNpZW50IHNjb3JlIGVxdWF0aW9uLiBBcyBhIHJlc3VsdCwgVE1MRSBpcyBhICoqZG91YmxlIHJvYnVzdCBlc3RpbWF0b3IqKi4KNC4gKipUTUxFKiogaXQgd2lsbCBiZSBjb25zaXN0ZW50IGZvciAkXHBzaShQX3swfSkkIGlzIGVpdGhlciB0aGUgY29uZGl0aW9uYWwgZXhwZWN0YXRpb24gRSRfezB9JChZfEEsVykgb3IgdGhlIGV4cG9zdXJlIG1lY2hhbmlzbSBQJF97MH0kKEF8VykgYXJlIGVzdGltYXRlZCBjb25zaXN0ZW50bHkuIFdoZW4gYm90aCBmdW5jdGlvbnMgYXJlIGNvbnNpc3RlbnRseSBlc3RpbWF0ZWQsIHRoZSAqKlRNTEUqKiB3aWxsIGJlIGVmZmljaWVudCBpbiB0aGF0IGl0IGFjaGlldmVzIHRoZSBsb3dlc3QgYXN5bXB0b3RpYyB2YXJpYW5jZSBhbW9uZyBhIGxhcmdlIGNsYXNzIG9mIGVzdGltYXRvcnMuIFRoZXNlIGFzeW1wdG90aWMgcHJvcGVydGllcyB0eXBpY2FsbHkgdHJhbnNsYXRlIGludG8gbG93ZXIgYmlhcyBhbmQgdmFyaWFuY2UgaW4gZmluaXRlIHNhbXBsZXMuW0BidWgyMDE2XQo1LiBUaGUgYWR2YW50YWdlcyBvZiBUTUxFIGhhdmUgYmVlbiByZXBlYXRlZGx5IGRlbW9uc3RyYXRlZCBpbiBib3RoIHNpbXVsYXRpb24gc3R1ZGllcyBhbmQgYXBwbGllZCBhbmFseXNlcy5bQHZhbjIwMTFdCjYuIFRoZSBwcm9jZWR1cmUgaXMgYXZhaWxhYmxlIHdpdGggc3RhbmRhcmQgc29mdHdhcmUgc3VjaCBhcyB0aGUgKip0bWxlKiogcGFja2FnZSBpbiBSIFtAZ3J1YmVyMjAxMV0uCgojVGhlIEctRm9ybXVsYQoxLiAkXHBzaShQX3swfSlcLD1cLFxzdW1fe3d9XCxcbGVmdFtcc3VtX3t5fVwsUChZPXlcbWlkIEE9MSxXPXcpLVwsXHN1bV97eX1cLFAoWSA9IHlcbWlkIEE9MCxXPXcpXHJpZ2h0XVAoVz13KSQgIAp3aGVyZSAgCiRQKFkgPSB5IFxtaWQgQSA9IGEsIFcgPSB3KVwsPVwsXGZyYWN7UChXID0gdywgQSA9IGEsIFkgPSB5KX17XHN1bV97eX1cLFAoVyA9IHcsIEEgPSBhLCBZID0geSl9JCAgCmlzIHRoZSBjb25kaXRpb25hbCBwcm9iYWJpbGl0eSBkaXN0cmlidXRpb24gb2YgWSA9IHksIGdpdmVuIEEgPSBhLCBXID0gdyBhbmQsICAKJFAoVyA9IHcpXCw9XCxcc3VtX3t5LGF9XCxQKFcgPSB3LCBBID0gYSwgWSA9IHkpJCAgCjIuIFVzaW5nIGNsYXNzaWNhbCByZWdyZXNzaW9uIG1ldGhvZHMgdG8gY29udHJvbCBjb25mb3VuZGluZyByZXF1aXJlcyBtYWtpbmcgdGhlIGFzc3VtcHRpb24gdGhhdCB0aGUgZWZmZWN0IG1lYXN1cmUgaXMgY29uc3RhbnQgYWNyb3NzIGxldmVscyBvZiBjb25mb3VuZGVycyBpbmNsdWRlZCBpbiB0aGUgbW9kZWwuCjMuIEFsdGVybmF0aXZlbHksICoqc3RhbmRhcmRpemF0aW9uKiogYWxsb3dzIHVzIHRvIG9idGFpbiBhbiB1bmNvbmZvdW5kZWQgc3VtbWFyeSBlZmZlY3QgbWVhc3VyZSB3aXRob3V0IHJlcXVpcmluZyB0aGlzIGFzc3VtcHRpb24uVGhlICoqRy1mb3JtdWxhKiogaXMgYSAqZ2VuZXJhbGl6YXRpb24gb2Ygc3RhbmRhcmRpemF0aW9uKltAcm9iaW5zMTk4Nl0KCiNDYXVzYWwgYXNzdW1wdGlvbnMgClVuZGVyIHRoZSBjb3VudGVyZmFjdHVhbCBmcmFtZXdvcmssIHdlIGhhdmUgdG8gY29uc2lkZXIgdGhlIGZvbGxvd2luZyBhc3N1bXB0aW9ucyB0byBjb25zaWRlciB0aGUgZXN0aW1hdGUgb2YgdGhlIEFURSBhcyBjYXVzYWw6IAoKIyNDTUkgb3IgUmFuZG9taXphdGlvbiAKKCRZX3swfSxZX3sxfVxwZXJwJEF8Vykgb2YgdGhlIGJpbmFyeSB0cmVhdG1lbnQgZWZmZWN0IChBKSBvbiB0aGUgb3V0Y29tZSAoWSkgZ2l2ZW4gdGhlIHNldCBvZiBvYnNlcnZlZCBjb3ZhcmlhdGVzIChXKSwgd2hlcmUgVyA9IChXMSwgIFcyLCBXMywg4oCmICwgV2spLiAKCiMjUG9zaXRpdml0eSAKYSDPtSBBOiBQKEE9YSB8IFcpID4gMCAgClAoQT0xfFc9dykgPiAwIGFuZCBQKEE9MHwgVyA9IHcpID4gMCBmb3IgZWFjaCBwb3NzaWJsZSB3LiAgCgojI0NvbnNpc3RlbmN5IG9yIFNVVFZBOiAKVGhlIG9ic2VydmVkIG91dGNvbWUgdmFsdWUsIHVuZGVyIHRoZSBvYnNlcnZlZCB0cmVhdG1lbnQsIGlzIGVxdWFsIHRvIHRoZSBjb3VudGVyZmFjdHVhbCBvdXRjb21lIGNvcnJlc3BvbmRpbmcgdG8gdGhlIG9ic2VydmVkIHRyZWF0bWVudCBmb3IgaWRlbnRpY2FsIGluZGVwZW5kZW50IGRpc3RyaWJ1dGVkIChpLmkuZC4pIHZhcmlhYmxlcy4gICAgCgojVE1MRSBmbG93IGNoYXJ0IAoqKlNvdXJjZSoqIDoJTWFyayB2YW4gZGVyIExhYW4gYW5kIFNoZXJyaSBSb3NlLiBUYXJnZXRlZCBsZWFybmluZzogY2F1c2FsIGluZmVyZW5jZSBmb3Igb2JzZXJ2YXRpb25hbCBhbmQgZXhwZXJpbWVudGFsIGRhdGFTcHJpbmdlciBTZXJpZXMgaW4gU3RhdGlzdGljcywgMjAxMQohW10oRmlndXJlcy90bWxlLnBuZykKCiNEYXRhIGdlbmVyYXRpb24KSW4gUiB3ZSBjcmVhdGUgYSBmdW5jdGlvbiB0byBnZW5lcmF0ZSB0aGUgZGF0YSB3aXRoIHRoZSBpbnB1dCBudW1iZXIgb2YgZHJhd3MgYW5kIHRoZSBvdXRwdXQgdGhlIG9ic2VydmVkIGRhdGEgKE9ic0RhdGEpIHBsdXMgdGhlIGNvdW50ZXJmYWN0dWFscyAoWTEsIFkwKS4gICAKVGhlIG9ic2VydmVkIGRhdGE6ICAKMS4gWTogbW9ydGFsaXR5IGJpbmFyeSBpbmRpY2F0b3IgKDEgZGVhdGgsIDAgYWxpdmUpIAoyLiBBOiBiaW5hcnkgdHJlYXRtZW50IGZvciBlbWVyZ2VuY3kgcHJlc2VudGF0aW9uIGF0IGNhbmNlciBkaWFnbm9zaXMgICgxIEVQLCAwIE5vbkVQKSAgICAKMy4gVzE6IEdlbmRlciAoMSBtYWxlOyAwIGZlbWFsZSkgIAo0LiBXMjogQWdlIGF0IGRpYWdub3NpcyAoMCA8NjU7IDEgPj02NSkgIAo0LiBXMzogQ2FuY2VyIFROTSBjbGFzc2lmaWNhdGlvbiAoc2NhbGUgZnJvbSAxIHRvIDQpICAKNS4gVzQ6IENvbW9yYmlkaXRpZXMgKHNjYWxlIGZyb20gMSB0byA1KSAgCgpgYGB7cn0KI2luc3RhbGwucGFja2FnZXMoImJyb29tIikKb3B0aW9ucyhkaWdpdHM9MykKZ2VuZXJhdGVEYXRhIDwtIGZ1bmN0aW9uKG4pewogIHcxIDwtIHJiaW5vbShuLCBzaXplPTEsIHByb2I9MC41KQogIHcyIDwtIHJiaW5vbShuLCBzaXplPTEsIHByb2I9MC42NSkKICB3MyA8LSBydW5pZihuLCBtaW49MCwgbWF4PTQpCiAgdzQgPC0gcnVuaWYobiwgbWluPTAsIG1heD01KQogIEEgIDwtIHJiaW5vbShuLCBzaXplPTEsIHByb2I9IHBsb2dpcygtMC40ICsgMC4yKncyICsgMC4xNSp3MyArIDAuMip3NCkpCiAgWSAgPC0gcmJpbm9tKG4sIHNpemU9MSwgcHJvYj0gcGxvZ2lzKC0xICsgQSAtMC4xKncxICsgMC4zKncyICsgMC4yNSp3MyArIDAuMip3NCkpCiAgCiAgIyBjb3VudGVyZmFjdHVhbAogIFkuMSA8LSByYmlub20obiwgc2l6ZT0xLCBwcm9iPSBwbG9naXMoLTEgKyAxIC0wLjEqdzEgKyAwLjMqdzIgKyAwLjI1KnczICsgMC4yKnc0KSkKICBZLjAgPC0gcmJpbm9tKG4sIHNpemU9MSwgcHJvYj0gcGxvZ2lzKC0xICsgMCAtMC4xKncxICsgMC4zKncyICsgMC4yNSp3MyArIDAuMip3NCkpCiAgCiAgIyByZXR1cm4gZGF0YS5mcmFtZQogIGRhdGEuZnJhbWUodzEsIHcyLCB3MywgdzQsIEEsIFksIFkuMSwgWS4wKQp9CnNldC5zZWVkKDc4NTgpCk9ic0RhdGEgPC0gZ2VuZXJhdGVEYXRhKG49MTAwMCkKVHJ1ZV9Qc2kgPC0gbWVhbihPYnNEYXRhJFkuMS1PYnNEYXRhJFkuMCk7VHJ1ZV9Qc2kKQmlhc19Qc2kgPC0gc3VtbWFyeShsbShkYXRhPU9ic0RhdGEsIFl+QSkpO0JpYXNfUHNpJGNvZWYKYGBgCgojRGF0YSB2aXN1YWxpemF0aW9uCmBgYHtyfQojIERUIHRhYmxlID0gaW50ZXJhY3RpdmUKIyBpbnN0YWxsLnBhY2thZ2VzKCJEVCIpICMgaW5zdGFsbCBEVCBmaXJzdApsaWJyYXJ5KERUKQpkYXRhdGFibGUoaGVhZChPYnNEYXRhLCBuID0gbnJvdyhPYnNEYXRhKSksIG9wdGlvbnMgPSBsaXN0KHBhZ2VMZW5ndGggPSA1KSkgCmBgYAoKI1RNTEUgaW1wbGVtZW50YXRpb24KCiMjMXN0IHN0ZXA6IEUkX3swfSQoWXxBLFcpICAKRXN0aW1hdGlvbiBvZiB0aGUgaW5pdGlhbCBwcm9iYWJpbGl0eSBvZiB0aGUgb3V0Y29tZSAoWSkgZ2l2ZW4gdGhlIHRyZWF0bWVudCAoQSkgYW5kIHRoZSBzZXQgb2YgY292YXJpYXRlcyAoVyksIGRlbm90ZWQgYXMgdGhlICRRX3swfSQoQSwqKlcqKikuIFRvIGVzdGltYXRlICRRX3swfSQoQSwqKlcqKikgd2UgY2FuIHVzZSBhIHN0YW5kYXJkIGxvZ2lzdGljIHJlZ3Jlc3Npb24gbW9kZWw6IAoKJGxvZ2l0W1AoWT0xfEEsVyldXCw9XCxcYmV0YV97MH1cLCtcLFxiZXRhX3sxfUFcLCtcLFxiZXRhX3syfV57VH0kKipXKiouICAgIAoKVGhlcmVmb3JlLCB3ZSBjYW4gZXN0aW1hdGUgdGhlIGluaXRpYWwgcHJvYmFiaWxpdHkgKGFzIGZvbGxvd3M6ICAuICAgICAgKDEpClRoZSBwcmVkaWN0ZWQgcHJvYmFiaWxpdHkgY2FuIGJlIGVzdGltYXRlZCB1c2luZyB0aGUgU3VwZXIgTGVhcm5lciBsaWJyYXJ5IGltcGxlbWVudGVkIGluIHRoZSBSIHBhY2thZ2Ug4oCcU3VwZXItTGVhcm5lcuKAnTYgdG8gaW5jbHVkZSBhbnkgdGVybXMgdGhhdCBhcmUgZnVuY3Rpb25zIG9mIEEgb3IgVyAoZS5nLiwgcG9seW5vbWlhbCB0ZXJtcyBvZiBBIGFuZCBXLCBhcyB3ZWxsIGFzIHRoZSBpbnRlcmFjdGlvbiB0ZXJtcyBvZiBBIGFuZCBXLCBjYW4gYmUgY29uc2lkZXJlZCkuIENvbnNlcXVlbnRseSwgZm9yIGVhY2ggc3ViamVjdCwgdGhlIHByZWRpY3RlZCBwcm9iYWJpbGl0aWVzIGZvciBib3RoIHBvdGVudGlhbCBvdXRjb21lcyAgYW5kICBjYW4gYmUgZXN0aW1hdGVkIGJ5IHNldHRpbmcgQSA9IDAgYW5kIEEgPSAxIGZvciBldmVyeW9uZSByZXNwZWN0aXZlbHk6CiBhbmQsLgoKCgojIFRoYW5rIHlvdSAgClRoYW5rIHlvdSBmb3IgcGFydGljaXBhdGluZyBpbiB0aGlzIHR1dG9yaWFsLiAgCklmIHlvdSBoYXZlIHVwZGF0ZXMgb3IgY2hhbmdlcyB0aGF0IHlvdSB3b3VsZCBsaWtlIHRvIG1ha2UsIHBsZWFzZSBzZW5kIDxhIGhyZWY9Imh0dHBzOi8vZ2l0aHViLmNvbS9taWdhcmlhbmUvTUFMRiIgdGFyZ2V0PSJfYmxhbmsiPm1lPC9hPiBhIHB1bGwgcmVxdWVzdC4KQWx0ZXJuYXRpdmVseSwgaWYgeW91IGhhdmUgYW55IHF1ZXN0aW9ucywgcGxlYXNlIGUtbWFpbCBtZS4gIAoqKk1pZ3VlbCBBbmdlbCBMdXF1ZSBGZXJuYW5kZXoqKiAgCioqRS1tYWlsOioqICptaWd1ZWwtYW5nZWwubHVxdWUgYXQgbHNodG0uYWMudWsqICAKKipUd2l0dGVyKiogYEBXQVRaSUxFSWAgIAoKIyBTZXNzaW9uIEluZm8gCmBgYHtyIHNlc3Npb24taW5mbywgcmVzdWx0cyA9J21hcmt1cCd9CmRldnRvb2xzOjpzZXNzaW9uX2luZm8oKQpgYGAKIyBSZWZlcmVuY2VzIAo=